Reliability based keyframe switching system and method adaptable to icp

ABSTRACT

A reliability based keyframe switching system adaptable to iterative closest point (ICP) includes a camera that captures frames; a matching device that determines corresponding pairing between the frames for an ICP operation to form a set of plural point-pairs; a transformation device that performs transformation estimation to estimate transformation that minimizes distances of the point-pairs, and determines whether the estimated transformation converges; and a reliability device that determines whether the ICP operation is reliable, and replaces a current keyframe with a new keyframe if the ICP operation is determined to be unreliable.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention generally relates to visual odometry, and moreparticularly to a reliability based keyframe switching method adaptableto iterative closest point (ICP).

2. Description of Related Art

Visual odometry is a process adopted in robotics and computer vision todetermine position and orientation of a robot by analyzing associatedcamera images, for example, captured by a RGB-D camera. The motion ofthe robot may be estimated by aligning a source to a target RGB-D frameusing an iterative closest point (ICP) method. FIG. 1 shows a blockdiagram illustrating a conventional visual odometry system 100 using ICPas disclosed in “Fast Visual Odometry Using Intensity-Assisted IterativeClosest Point,” entitled to Shile Li et al., July 2016, IEEE ROBOTICSAND AUTOMATION LETTERS, VOL. 1, NO. 2, the disclosure of which isincorporated herein by reference.

The visual odometry system 100 is proposed to reduce computational cost,and reduce influence of outliers and noises. In the visual odometrysystem 100 of FIG. 1, salient point selection 11 is performed on thesource frame, where points that provide valuable information for ICP areselected. The search of correspondences 12 is performed, where thematching point is determined. Weighting of corresponding pairs 13 isperformed based on robust static. Incremental transformation 14 isperformed to minimize the distances between the establishedcorrespondences. The above operations 11-14 are performed iterativelyuntil the incremental transformation is smaller than a threshold or themaximum allowable iteration number has reached.

Conventional frame-to-frame alignment method inherently accumulatesdrift, because there is always a small error caused by sensor noise inthe estimate. In order to overcome the drift problem, Christian Kerl etal. discloses “Dense Visual SLAM for RGB-D Cameras,” 2013, Proc. of theInt. Conf. on Intelligent Robot Systems (IROS), the disclosure of whichis incorporated herein by reference. Keyframe-based pose simultaneouslocalization and mapping (SLAM) method is adopted herein to limit localdrift by estimating transformation between the current image and akeyframe. As long as the camera stays close enough to the keyframe, nodrift is accumulated. The SLAM system needs to additionally performkeyframe selection, loop closure detection and validation, and mapoptimization.

A need has thus arisen to propose a novel method adaptable to ICP toovercome drawbacks of conventional schemes.

SUMMARY OF THE INVENTION

In view of the foregoing, it is an object of the embodiment of thepresent invention to provide a reliability based keyframe switchingmethod for substantially enhancing performance of iterative closestpoint (ICP).

According to one embodiment, a reliability based keyframe switchingsystem adaptable to iterative closest point (ICP) includes a camera, amatching device, a transformation device and a reliability device. Thecamera captures frames. The matching device determines correspondingpairing between the frames for an ICP operation to form a set of pluralpoint-pairs. The transformation device performs transformationestimation to estimate transformation that minimizes distances of thepoint-pairs, and determines whether the estimated transformationconverges. The reliability device determines whether the ICP operationis reliable, and replaces a current keyframe with a new keyframe if theICP operation is determined to be unreliable.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram illustrating a conventional visual odometrysystem using ICP;

FIG. 2A shows a flow diagram illustrating a reliability based keyframeswitching method adaptable to iterative closest point (ICP) according toone embodiment of the present invention;

FIG. 2B shows a block diagram illustrating a reliability based keyframeswitching system adaptable to iterative closest point (ICP) according toone embodiment of the present invention;

FIG. 3 shows a detailed flow diagram of step 25 of FIG. 2A;

FIG. 4A shows an exemplary vector of keypoint that is projected on thedirection of a frame vector;

FIG. 4B shows exemplary distribution of projection in the strengthhistogram; and

FIG. 5 shows a flow diagram illustrating a reliability based keyframeswitching method adaptable to iterative closest point (ICP) according toanother embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 2A shows a flow diagram illustrating a reliability based keyframeswitching method 200 adaptable to iterative closest point (ICP)according to one embodiment of the present invention, and FIG. 2B showsa block diagram illustrating a reliability based keyframe switchingsystem 300 adaptable to iterative closest point (ICP) according to oneembodiment of the present invention. The steps of the reliability basedkeyframe switching method (“method” hereinafter) 200 and the blocks ofthe reliability based keyframe switching system (“system” hereinafter)300 may be implemented by electrical circuits, computer software ortheir combination. For example, at least a portion of the method 200 andthe system 300 may be performed in a digital image processor. In anotherexample, at least a portion of the method 200 and the system 300 may beimplemented by an instruction-controlled computer. In one exemplaryembodiment, the method 200 and the system 300 may be adapted to anaugmented reality (AR) device. Hardware components for the AR device mayprimarily include a processor (e.g., an image processor), a display(e.g., head-mounted display) and sensors (e.g., a color-depth camerasuch as RGB-D camera for red, green, blue plus depth). Specifically, thesensors or camera captures scenes to generate image frames (or simplyframes), which are then fed to the processor that performs theoperations of the method 200 and the system 300. Augmented reality isthen rendered in the display.

In step 21, (image) frames captured by a camera such as athree-dimensional (3D) camera may be inputted. In the specification, a3D camera (e.g., RGB-D camera) is a camera capable of capturing a depth(D) image in addition to a color (e.g., red (R), green (G) and blue (B))image.

In step 22, corresponding pairing between the frames (e.g., between acurrent frame and a previous frame) for an ICP operation is determinedby a matching device 32. Specifically, for each point on a source pointset P, corresponding closest point may be found on a target point set Q,thereby forming a set of plural point-pairs (p_(i),q_(i)). Subsequently,weighting of the point-pairs may be performed.

In step 23, transformation estimation may be performed by atransformation device 33 to estimate transformation, composed of arotation and a translation, that minimize distances of the point-pairs.In step 24, if the estimated transformation converges (e.g., therotation and the translation are less than predetermined thresholdsrespectively), the flow goes to step 25. If the estimated transformationdoes not converge, the flow goes back to step 22. Details of ICPconvergence may be referred to “Registration with the Point CloudLibrary: A Modular Framework for Aligning in 3-D,” entitled to Dirk Holzet al., Dec. 1, 2015, IEEE Robotics & Automation Magazine, thedisclosures of which are incorporated herein by reference.

In step 25, reliability for the ICP operation may be determined by areliability device 34. FIG. 3 shows a detailed flow diagram of step 25of FIG. 2A. Specifically, in step 251, a strength histogram may beobtained by projecting vectors of keypoints of a (current) keyframe on adirection of a frame vector obtained from step 23, thereby obtainingprojected components or strengths constructing the strength histogram,details of which may be referred to “Geometrically stable sampling forthe ICP algorithm,” entitled to Natasha Gelfand et al., October 2003,Fourth International Conference on 3-D Digital Imaging and Modeling, thedisclosures of which are incorporated herein by reference. FIG. 4A showsan exemplary vector of keypoint Ti that is projected on the direction ofa frame vector T*. FIG. 4B shows exemplary distribution of projection inthe strength histogram.

In step 252, a strength index may be obtained based on the strengthhistogram. In one embodiment, the strength index may be an average valueof the (absolute) projected components of the strength histogram. Inanother embodiment, the strength index may be a variance value of theprojected components of the strength histogram. It is noted that thegreater the strength index (e.g., average or variance value) is, themore reliable the ICP is. In step 253, if the strength index is greaterthan a predetermined threshold (e.g., 0.2 of the average value), areliable state is determined, otherwise an unreliable state isdetermined.

Referring back to the method 200 of FIG. 2A, if the unreliable state isdetermined in step 25, a new keyframe (e.g., a preceding frame) is usedto replace the current keyframe in step 26. Subsequently, in step 27, aregion of interest (ROI) is detected and points are sampled therefrom.Next, the flow goes back to step 22 to execute another iteration of ICPoperation. If the reliable state is determined in step 25, the estimatedtransformation (from step 23) may be outputted. Details of the ICPoperation may be referred to the aforementioned “Fast Visual OdometryUsing Intensity-Assisted Iterative Closest Point” and “Dense Visual SLAMfor RGB-D Cameras.” Further details of the ICP operation may be referredto “Multiview Registration for Large Data Sets,” entitled to Kari Pulli,October 1999, Second International Conference on 3D Digital Imaging andModeling; and “Tracking a Depth Camera: Parameter Exploration for FastICP,” entitled to Francois Pomerleau et al., September 2011, IEEE/RSJInternational Conference on Intelligent Robots and Systems, thedisclosures of which are incorporated herein by reference.

FIG. 5 shows a flow diagram illustrating a reliability based keyframeswitching method 500 adaptable to iterative closest point (ICP)according to another embodiment of the present invention. Thereliability based keyframe switching method (“method” hereinafter) 500is similar to the method 200 (FIG. 2A) except that the order of step 24and step 25 is reversed. Specifically, after transformation is estimatedin step 23, no matter the estimated transformation converges or not, theflow goes to step 25 to determine reliability for ICP. If the unreliablestate is determined in step 25, a new keyframe (e.g., a preceding frame)is used to replace the current keyframe in step 26. If the reliablestate is determined in step 25, the flow goes to step 24 to determinewhether the estimated transformation converges. If the estimatedtransformation converges (e.g., the rotation and the translation areless than predetermined thresholds respectively), the estimatedtransformation is then outputted. If the estimated transformation doesnot converge, the flow goes back to step 22.

Although specific embodiments have been illustrated and described, itwill be appreciated by those skilled in the art that variousmodifications may be made without departing from the scope of thepresent invention, which is intended to be limited solely by theappended claims.

What is claimed is:
 1. A reliability based keyframe switching systemadaptable to iterative closest point (ICP), comprising: a camera thatcaptures frames; a matching device that determines corresponding pairingbetween the frames for an ICP operation to form a set of pluralpoint-pairs; a transformation device that performs transformationestimation to estimate transformation that minimizes distances of thepoint-pairs, and determines whether the estimated transformationconverges; and a reliability device that determines whether the ICPoperation is reliable, and replaces a current keyframe with a newkeyframe if the ICP operation is determined to be unreliable.
 2. Thesystem of claim 1, wherein the matching device further performsweighting on the point-pairs.
 3. The system of claim 1, wherein thetransformation device determines that the estimated transformationconverges when rotation and translation of the estimated transformationare less than predetermined thresholds, respectively.
 4. The system ofclaim 1, wherein the reliability device performs the following steps todetermine whether the ICP operation is reliable: obtaining a strengthhistogram by projecting vectors of keypoints of a current keyframe on adirection of a frame vector, thereby obtaining projected componentsconstructing the strength histogram; obtaining a strength index based onthe strength histogram; and if the strength index is greater than apredetermined threshold, a reliable state is determined, otherwise anunreliable state is determined.
 5. The system of claim 4, wherein thestrength index is an average value or a variance value of the projectedcomponents of the strength histogram.
 6. The system of claim 1, afterreplacing the current keyframe with the new keyframe, the reliabilitydevice further performs the following steps: detecting a region ofinterest (ROI); and sampling points from the ROI.
 7. A reliability basedkeyframe switching method adaptable to iterative closest point (ICP),comprising: capturing frames; determining corresponding pairing betweenthe frames for an ICP operation, thereby forming a set of pluralpoint-pairs; performing transformation estimation to estimatetransformation that minimizes distances of the point-pairs; determiningwhether the estimated transformation converges; determining whether theICP operation is reliable if the estimated transformation converges; andreplacing a current keyframe with a new keyframe if the ICP operation isdetermined to be unreliable.
 8. The method of claim 7, wherein the frameis captured by a three-dimensional (3D) camera.
 9. The method of claim7, further comprising: performing weighting on the point-pairs.
 10. Themethod of claim 7, wherein the estimated transformation converges whenrotation and translation of the estimated transformation are less thanpredetermined thresholds, respectively.
 11. The method of claim 7,wherein the step of determining whether the ICP operation is reliablecomprises: obtaining a strength histogram by projecting vectors ofkeypoints of a current keyframe on a direction of a frame vector,thereby obtaining projected components constructing the strengthhistogram; obtaining a strength index based on the strength histogram;and if the strength index is greater than a predetermined threshold, areliable state is determined, otherwise an unreliable state isdetermined.
 12. The method of claim 11, wherein the strength index is anaverage value or a variance value of the projected components of thestrength histogram.
 13. The method of claim 7, after replacing thecurrent keyframe with the new keyframe, further comprising: detecting aregion of interest (ROI); and sampling points from the ROI.
 14. Areliability based keyframe switching method adaptable to iterativeclosest point (ICP), comprising: capturing frames; determiningcorresponding pairing between the frames for an ICP operation, therebyforming a set of plural point-pairs; performing transformationestimation to estimate transformation that minimizes distances of thepoint-pairs; determining whether the ICP operation is reliable;determining whether the estimated transformation converges if the ICPoperation is determined to be reliable; and replacing a current keyframewith a new keyframe if the ICP operation is determined to be unreliable.15. The method of claim 14, wherein the frame is captured by athree-dimensional (3D) camera.
 16. The method of claim 14, furthercomprising: performing weighting on the point-pairs.
 17. The method ofclaim 14, wherein the estimated transformation converges when rotationand translation of the estimated transformation are less thanpredetermined thresholds, respectively.
 18. The method of claim 14,wherein the step of determining whether the ICP operation is reliablecomprises: obtaining a strength histogram by projecting vectors ofkeypoints of a current keyframe on a direction of a frame vector,thereby obtaining projected components constructing the strengthhistogram; obtaining a strength index based on the strength histogram;and if the strength index is greater than a predetermined threshold, areliable state is determined, otherwise an unreliable state isdetermined.
 19. The method of claim 18, wherein the strength index is anaverage value or a variance value of the projected components of thestrength histogram.
 20. The method of claim 14, after replacing thecurrent keyframe with the new keyframe, further comprising: detecting aregion of interest (ROI); and sampling points from the ROI.